Add env variables to configure manifest cache#2993
Add env variables to configure manifest cache#2993kris-gaudel wants to merge 4 commits intoapache:mainfrom
Conversation
|
@kevinjqliu @Fokko @rambleraptor Please lmk what you think! |
|
@kris-gaudel I have a few questions:
|
Fokko
left a comment
Fokko
left a comment
There was a problem hiding this comment.
Thanks @kris-gaudel for raising this. I think it is reasonable to make this configurable. But I think we should reduce the number of changes and simplify where possible. For example, do we need PYICEBERG_MANIFEST__CACHE__ENABLED or can we set PYICEBERG_MANIFEST__CACHE__SIZE to 0 instead.
| # Global cache for ManifestFile objects, keyed by manifest_path. | ||
| # This deduplicates ManifestFile objects across manifest lists, which commonly | ||
| # share manifests after append operations. | ||
| _manifest_cache: LRUCache[str, ManifestFile] = LRUCache(maxsize=128) |
There was a problem hiding this comment.
I think we can significantly simplify the code, and reduce the number of changes.
manifest_cache_size = Config().get_int("manifest-cache-size") or 128
_manifest_cache: LRUCache[str, ManifestFile] = LRUCache(maxsize=manifest_cache_size)To enable this we need to change PYICEBERG_MANIFEST__CACHE__SIZE to PYICEBERG_MANIFEST_CACHE_SIZE
There was a problem hiding this comment.
Thank you for the feedback, my intention with the cache manager was to create some abstracted logic for interacting with the cache, since I thought the two configuration parameters was the desired behaviour, I think these new commits will address your concerns
kris-gaudel
force-pushed
the
kris-gaudel/configurable-manifest-cache
branch
from
9ceb651 to
4cad1c3
Compare
kris-gaudel
force-pushed
the
kris-gaudel/configurable-manifest-cache
branch
from
f3b9f3d to
f411547
Compare
3 participants
Closes #2952
Rationale for this change
Through discussion in issue #2325, we realized that there was a memory leak in the manifest cache. PR #2951 fixed this memory leak, but we decided that it would be best for developer experience if we developers could configure the cache for their needs.
This includes the ability to disable/enable the manifest cache, and configure its size
Are these changes tested?
Yes - unit tests are included to verify these changes
Are there any user-facing changes?
Yes:
PYICEBERG_MANIFEST__CACHE__ENABLEDcan be used to enable/disable the cache, andPYICEBERG_MANIFEST__CACHE__SIZEcan be used to configure the size of the manifest cache